Deriving and Managing Data Products in an Environmental Observation and Forecasting System

Large-scale scientific workflows can perform many computationally intensive tasks and generate large volumes of derived data products. These systems pose many challenges to both creating and managing data products, including efficiently executing tasks and tracking data product lineage and metadata. In this paper we describe our experiences implementing an experimental data-product management system to address these challenges for the CORIE Environmental Observation and Forecasting System. We present a novel architecture to store both data products and the tasks that create them. Our system in addition supports tasks to automatically perform system maintenance, and enables dataintensive tasks to execute on multiple nodes of a Grid. We present several challenges to executing existing scientific workflows on a Grid, and propose several techniques to improve task scheduling in this environment. Preliminary performance results show the potential benefits of these techniques.